Densely packed electronic systems

ABSTRACT

Components selected from bare die, surface mount devices and stacked devices are assembled using flip chip assembly methods on a printed circuit board assembly (PCBA) with no components having a mounted height exceeding a preferred height. The preferred height may correspond with the components having the highest power rating, because the most effective thermal coupling to a heat sinking surface will then be provided to these high-power components. A blade server is configured with the back face of high-power components coupled to a metal tank carrying cooling water. An electronic system has laminate blocks comprising repeated laminations of PCBAs coupled to metal foils. The laminate blocks are coupled to heat sink surfaces in direct contact with cooling liquid. Power density is superior to existing high-performance computing (HPC) systems and data center servers.

TECHNICAL FIELD

This invention relates to the field of electronic system heatdissipation and more specifically to water cooled printed circuit boardassemblies, blade servers and electronic systems.

BACKGROUND

Rack mounted servers employ processors and supporting devices mounted onrigid printed circuit boards. High-power chips are usually sold aspackaged devices that are mounted on printed circuit boards. Thepackaged devices consume considerably more space than the original diecontained within the package. Mounting bare die on a printed circuitboard can provide a higher chip density but this requires improved heatdissipation.

In the prior art, a CRAY E5-4669v4 supercomputer uses up to 384 64-bitIntel Xeon processors running at 2.2 GHz, and each processor can includeup to 22 cores. The volume of the compute cabinet is 176,929 cubicinches, and the volume of the blower cabinet is 60,669 cubic inches fora total volume of 237,598 cubic inches. Power consumed by the computecabinet is 90 kW in the maximum configuration. The density of processorsper unit system volume is 384/237,598 in³ or 0.0016 processors/in³.

Also in the prior art is the HP PROLIANT BL460cG8 blade server. It hasdimensions of 2.2-inch wide, 7.1-inch high, and 20.4-inches deep. Itemploys dual INTEL OCTA-CORE XEON E5-2660v4 processors running at 2.2GHz. Each processor die has an area of 306.2 square millimeters and eachpackaged processor has an area of 2,362 square millimeters. The bladeserver uses a single printed circuit board having an area of 140 squareinches. The number of packaged processor chips per unit system volume is2/319 in³=0.0063 processors/in³.

Also in the prior art is a GOOGLE water-cooled device used in datacenters. Tensor processing unit (TPU v3) incorporates applicationspecific processors tailored for acceleration of artificial intelligence(AI) applications. A third-generation device was announced on May 8,2018. Tubes carry cooling water to packages comprising high powerdevices. The packaged devices are mounted on a printed circuit board.

There is a need in the art for server electronics having increasedspatial and power densities, wherein the thermal environment isconfigured to support higher levels of cooling power.

SUMMARY

In accordance with a first aspect of the invention, a printed circuitboard assembly (PCBA) includes electronic components assembled on asubstrate using flip chip assembly methods. Using a wide variety ofassembly options, it can be arranged that none of the components mountedon the PCBA have a height greater than a preferred height. Theelectronic components mounted on the substrate may be selected from baredie, surface mount devices, and stacked devices as non-limitingexamples. Stacked devices may comprise an interposer or a chiplet or anembedded multi-die interconnection bridge (EMIB). A planarizing fillermay be disposed between the components of the PCBA. The outer envelopeof the PCBA may have the shape of a rectangular prism, with the backside of mounted devices at or near the outer envelope for the mosteffective cooling; this geometric shape may be particularly apparentwhen the PCBA is filled with the planarizing filler. The substrate mayinclude traces that connect with terminals of a PCBA connector.

In accordance with a second aspect of the invention a lamination may beformed by coupling the PCBA with a metal member using a thermalinterface material. When the PCBA is coupled with the metal member, theback face of a high-power component having the preferred mounted heightwill have a short thermal path to a heat sinking surface (the metalmember), wherein the separation between them comprises only a thin layerof thermal interface material. Thus, the cooling performance will beoptimized for the high-power component.

In accordance with a third aspect of the invention a method forassembling a printed circuit board assembly is described. The methodincludes the steps of: providing a printed circuit board, selectingcomponents to be mounted on the printed circuit board from bare die,surface mount devices, and stacked devices, mounting the selectedcomponents on the printed circuit board using flip chip assemblymethods, and thermally coupling back faces of the selected components toa heat sinking surface using a thermal interface material. The selectedcomponents may be ranked according to their rated power and assemblymethods may be selected for assembling the ranked components whereinback faces of selected components having a higher rated power havepreferably a higher height when mounted on the printed circuit boardthan back faces of selected components having a lower rated power. Theselection of components and assembly methods may be iterated to improvea correlation of component power to component height. A filler may bedisposed between and atop the selected components such as to cover allof the selected components. A grinding or a polishing process may beapplied to remove filler material and semiconductor material as requiredto achieve a polished planar surface having at least some of the backfaces of selected components exposed, prior to thermally coupling thepolished planar surface to the heat sinking surface using thermalinterface material.

In accordance with a fourth aspect of the invention, a blade servercomprises a printed circuit board assembly (PCBA) thermally coupled to ametal tank. The PCBA comprises a plurality of bare die rather thanpackaged devices. The bare die may be provided with flip chip terminals.Surface mount devices (SMDs) may also be mounted on the PCBA. Stackeddevices that may comprise an interposer or a chiplet may also be mountedon the PCBA. A filler may serve to planarize the PCBA, filling holes orgaps around the components. The mounted components are cooled by bondingthe back side of each component against a wall of a tank in which liquidcoolant is circulated. Depending on the height of a mounted component, athin layer of filler material may cover the back side of the component;however, at the preferred mounting height there will be no covering offiller material and cooling will be optimized. The tank has a waterinput and a water output for circulating coolant water. The PCBA mayinclude at least one connector for connecting external signals andpower. The blade server may have a rated power consumption exceeding 16watts per cubic inch of blade server volume.

In accordance with a fifth aspect of the invention, an electronic systemcomprises an inner structure, wherein the inner structure includesrepeated laminations, each lamination comprising a PCBA and a metalfoil. Each PCBA may include a PCBA connector having terminals that arecoupled to corresponding terminals of a front-panel connector or arear-panel connector which may provide external signals and power. Thelaminations may be assembled into one or more laminate blocks. The metalfoil may be an alloy of copper. The PCBA may include bare die that areflip chip mounted on each side of a printed circuit board (PCB). Thebare die may include processor die and may further include memory die,communication-related die, power-related die, or any other die. Surfacemount devices (SMDs) and stacked devices may also be mounted on the PCB.Stacked devices may comprise an interposer or a chiplet or an embeddedmulti-die interconnection bridge (EMIB). The PCB may be flexible (a flexcircuit) and may include a looped portion for mounting at least oneconnector whose terminals are connected to corresponding terminals of afront or rear panel connector, for connecting external signals andpower. The electronic system may be configured with an outer tankenclosure having water inputs and outputs and may operate while theinner structure is substantially immersed in water. The outer tankenclosure may have dimensions of approximately 19 inches wide, 17.5inches high, and 36 inches long. A base of the inner structure mayinclude extended edges of metal foils that are connected to a baseplate. Each end of each block of the inner structure may include an endplate to which extended edges of copper foils are connected. Theconnections to the base plate and the end plates may be solderedconnections. The base plate and the end plates effectively seal againstwater intrusion into the inner structure. The baseplate may havesupporting fins for supporting the inner structure within the outer tankenclosure. The electronic system may include one or more inner tanksdisposed between laminate blocks and configured to carry water or otherliquid coolant. The electronic system may support a total powerconsumption exceeding 150 watts per cubic inch of system volume, whilemaintaining junction temperatures in the PCBA mounted components at asafe temperature, not to exceed 150° C. for example.

In accordance with a sixth aspect of the invention a method formanufacturing and deploying an electronic system comprises fabricatingan inner structure within an outer tank wherein the inner structurecomprises at least one laminate block and each laminate block comprisesa repeated lamination of a printed circuit board assembly and a metalfoil. For each repeated lamination, the metal foil is coupled to a heatsinking surface. A liquid coolant is circulated in passages providedbetween the inner structure and the outer tank, including circulation ofcoolant past the heat sinking surface. One or more inner tanksconfigured for circulating coolant may be disposed between laminateblocks. The electronic system may be coupled to external signals andpower through a front or rear panel connector.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate prior art and examples ofembodiments. The examples of embodiments, together with the descriptionof example embodiments, serve to explain the principles andimplementations of the embodiments.

FIG. 1 (Prior Art) is a front view of a CRAY XC040 supercomputer.

FIG. 2 (Prior Art) is a perspective view of an HP PROLIANT BL460cG8blade server.

FIG. 3 (Prior Art) is a perspective view of a printed circuit boardassembly having four die packages (tensor processing units, GOOGLE TPUs)cooled with tube delivered water.

FIG. 4 (Prior Art) depicts a TPU v4 containing up to 1,000 TPU cores.

FIG. 5 is a cross-sectional view of a blade server in accordance with anembodiment of the present disclosure.

FIG. 6 (prior art) is a partial layout of the printed circuit board inan HP PROLIANT BL460cG8 blade server.

FIG. 7 is a partial layout of the PCB of FIG. 6 using mounted bare dieinstead of packaged parts in accordance with an embodiment of thepresent disclosure.

FIG. 8 is an alternative partial layout of the PCB of FIG. 6 usingmounted bare die instead of packaged parts in accordance with anembodiment of the present disclosure.

FIG. 9 is a perspective view of a PCBA having the shape of a rectangularprism in accordance with an embodiment of the present disclosure.

FIG. 10 illustrates an electronic system in accordance with anembodiment of the present disclosure.

FIG. 11 is a cross-sectional view of the electronic system of FIG. 10indicated by section AA in accordance with an embodiment of the presentdisclosure.

FIG. 12 is an expanded cross-sectional view of a laminate structure inaccordance with an embodiment of the present disclosure.

FIG. 13 illustrates at full scale the layout of a processor group orchiplet including one processor die and 76 memory die in accordance withan embodiment of the present disclosure.

FIG. 14 is a flow chart of an exemplary process for manufacturing anddeploying an electronic system in accordance with an embodiment of thepresent disclosure.

FIG. 15 is a graph of cooling power versus filler thickness for theINTEL OCTA-CORE XEON E5-2660v4 processor chip in accordance with anembodiment of the present disclosure.

DETAILED DESCRIPTION

In embodiments of the present disclosure printed circuit boardassemblies (PCBAs) comprise mounted components selected from bare die,surface mount devices, and stacked devices. A stacked device may includean interposer or a chiplet or an embedded multi-die interconnectionbridge (EMIB). A stacked device may also include an organic substratehaving a redistribution layer; this may be described as “fan-out RDL”.Components within stacked devices may be interconnected using throughsilicon vias (TSVs) or embedded multi-die interconnect bridges (EMIBs).Other heterogeneous integration structures may be used, providing theback faces of active die are presented to a heat sinking surface inaccordance with a preferred height strategy. All components mounted on aPCBA of the present disclosure have a preferred maximum height,described herein as the preferred height. Preferably the highest powercomponents are mounted at or near the preferred height, because thatmounted height corresponds with the most effective thermal coupling to aheat sinking surface. An algorithm may be used to predetermine thepreferred height, wherein the components to be used are first rankedaccording to their power rating, then matching assembly techniquesselected, iterating as required. In some applications, processors willbe the components having the highest power rating, and they may alsorequire a stacked structure because of fan-out considerations, so thepreferred height may correspond to a stacked processor component. Anysuitable preferred height may be used. The preferred height may be anyvalue or range of values selected between about 0.5 mm and 3.0 mm. Itwill be appreciated that the preferred height may be less than 0.5 mm orgreater than 3.0 mm. In one particular example, the preferred height is2.5 mm.

A blade server is configured with the back face of high-power componentscoupled to a metal tank carrying cooling water. An electronic system haslaminate blocks comprising repeated laminations of PCBAs coupled tometal foils. The laminate blocks are thermally coupled to heat sinksurfaces in direct contact with cooling liquid. Power density issuperior to existing high-performance computing (HPC) systems and datacenter servers.

Prior art is discussed in reference to FIGS. 1-4 and FIG. 6. FIG. 1depicts the front side of a CRAY XC040 supercomputer 10 that utilizes aserver rack 11 as shown. The model E5-4669v4 supercomputer uses up to384 64-bit INTEL XEON processors running at 2.2 GHz, and each processorcan include up to 22 cores. The volume of the compute cabinet is 176,929cubic inches, and the volume of the blower cabinet is 60,669 cubicinches for a total volume of 237,598 cubic inches. Power consumed by thecompute cabinet is 90 kW in the maximum configuration. The density ofprocessors per unit system volume is 384/237,598 in³ or 0.0016processors/in³, a computational density to be greatly exceeded usingembodiments of the present disclosure.

FIG. 2 shows an HP PROLIANT BL460cG8 blade server 20. It has dimensionsof 2.2-inch wide, 7.1-inch high, and 20.4-inches deep. It employs dualINTEL OCTA-CORE XEON E5-2660v4 processors running at 2.2 GHz. Eachprocessor die has an area of 306.2 square millimeters and each packagedprocessor has an area of 2,362 square millimeters, 7.7 times larger or7.7×. In embodiments of the present disclosure chip-on-board (COB) andchip-on-flex (COF) technologies may be used instead ofpackaged-device-on-board technology. In the case of blade server 20, anassembly and manufacturing process that uses die rather than packageddevices has an area advantage of 7.7× for each processor. The bladeserver uses a single printed circuit board having an area of 140 squareinches. The number of packaged processor chips per unit system volume is2/319 in³=0.0063 processors/in³.

FIG. 3 shows a water-cooled device 30 used in data centers. Device 30 isa tensor processing unit (TPU v3) implemented with application specificprocessors 31 tailored for acceleration of artificial intelligence (AI)applications. This third-generation device was announced on May 8, 2018.The tubes 32 circulate cooling water in and out of cavities in thedevice packages. The devices are mounted on a printed circuit board 33as shown.

FIG. 4 shows an array of TPU servers, TPU v4, comprising up to 1,000 TPUcores, as an example of a water-cooled data center configuration. Tubes32 for carrying water coolant are shown.

FIG. 5 shows a blade server embodiment 50 of the present disclosure incross-section. Blade server 50 includes a pair of tanks 51 a and 51 bcontaining a coolant liquid 52 a, 52 b, each tank having a coolant input53 and a coolant output 54 for circulating the liquid. The tanks may becomprised of a copper alloy for good thermal conductivity, but anymaterial having high thermal conductivity and compatibility with thechosen coolant may be used. Water may be the chosen coolant for its lowcost and good thermal properties. A mixture of water and ethylene glycolmay also be used for the coolant. A printed circuit board (PCB) 55 isshown, with flip chip mounted die such as 56 a, 56 b, a surface mounteddevice 57, and a stacked device 58. Die 56 a is shown with copper pillarterminals 56 c. Die 56 b is shown with solder ball terminals 56 d. Anytype of flip chip terminal may be used. Stacked device 58 may comprisean interposer 59 as shown. A powerful processor may require thousands ofinput/output pins to be connected to the PCB 55. Accordingly, interposer59 may be used to redistribute a tight pad spacing on a mounted die suchas 60 a or 60 b to a more relaxed pad spacing on PCB 55 as shown in FIG.5. Interposer 59 may be configured as a chiplet substrate, carryingseveral interconnected chips. Since interposer 59 may be thin, ˜300 μmthick for example, in order for the height of a stacked device to beapproximately the same as the height of a silicon bare die it may beuseful to use a stiffer substrate material such as silicon carbide (SiC)instead of silicon (Si). The elastic modulus of SiC is around 410 GPa,compared with around 112 GPa for single crystal Si. The PCB 55 maycomprise an epoxy-glass laminate such as FR-4, or a flexible sheet ofKAPTON as non-limiting examples. PCB 55 with components mounted thereonbecomes a printed circuit board assembly (PCBA) 56. PCBA 56 may havecomponents mounted on one or both sides of PCB 55. PCBA 56 may include aplanarizing filler material 61 that may be applied by pouring or byscreen-printing for example. Filler material 61 is electricallynon-conductive and preferably has a high thermal conductivity. Anexample filler material 61 is SYLGARD 184, a filled silicone elastomeravailable from Dow, having a thermal conductivity of 0.27 W/m° K.SYLGARD 184 is self-leveling, enabling planarization of PCBA 56following assembly of the mounted components. It will be appreciatedthat other filler materials may be used. The mounted components mayinclude processors and memories and power-related devices as examples.Communication-related devices such as controllers for PCI Express or forGigabit Ethernet may be included. Surface mount devices (SMDs) such as57 may also be mounted on PCB 55; they may include capacitors andresistors and power-related components as examples. A preferred heightfor components mounted on a PCBA of the present disclosure may bepre-determined, based on desired cooling properties and assemblyconsiderations. Any component having less than the preferred height whenmounted on the PCBA may be used. However, components with mountedheights equal to or close to the preferred height will have respectivelythe best or close-to-the-best cooling properties, to be furtherdescribed in terms of the thermal resistance of specific diecombinations. Although an SMD may have a larger footprint than the sizeof a die embedded within it, the SMD footprint is usually small comparedwith that of a processor die, so its use may not substantially affectthe component density on a printed circuit board. It may be convenientfor cost reasons and time-to-market reasons to use SMDs rather thandevelop bare die equivalents. Traces of PCB 55 may connect to terminalsof a front panel connector 62 for access to external signals and power.

Printed circuit board assembly 56 is bonded on one side to a wall ofcooling tank 51 a using a die attach film (DAF) 63 a. It is bonded onthe other side to a wall of cooling tank 51 b using DAF 63 b. The DAF isused as a thermal interface material. A suitable die attach film isESP7666-HK-DAF available in thicknesses of 20 μm and 40 μm from AITechnology, and having a thermal conductivity of 1.8 W/m° K. Other dieattach films or pastes may be used, including ones filled with carbonnanotubes or other highly conductive materials for improved thermalconduction. Circuit traces of printed circuit board 55 connect toterminals of a front panel connector 62, providing access to externalsignals and power. An approximate width for this configuration of bladeserver 50 is 0.9 inches as shown, potentially compatible with ahalf-width blade server specification.

In FIG. 5 the back face of each mounted component is attached to awater-cooled tank with only a thin sheet of DAF in between. Thisprovides a favorable form factor for cooling the components, wherein thethermal path comprises an advantageous ratio of (heat sinking area) to(thermal path length).

Regarding the varying heights of assembled components, for the assemblymethod of the embodiments described herein it is desirable to select acombination of mounting methods that result in reasonably consistentheights among the wide variety of mountable components. When similarheights are selected, the rear face of each component will be close to aheat-sinking surface. The disposition of filler material 61 will besubstantially circumferential around the components, rather than addingthermal resistance between the rear face of a component, such as 64, andits corresponding heat sinking surface 65. 300 mm wafers may be around775 μm thick and 450 mm wafers may be around 950 μm thick. When dicedand prepared for assembly, flip chip terminals attached to the die maybe copper pillars with a height range of around 30-50 μm, or copperpillar bumps with a height range of 40-100 μm, or solder balls with adiameter range of 60-200 μm for flip chip applications, or 250-760 μmfor ball grid array (BGA) and fine pitch BGA applications. Additionally,wafers may be thinned to a desired thickness with a lower limit ofaround 50 μm. A preferred strategy is to first rank the systemcomponents according to their power consumption, each in its systemenvironment with respect to power-relevant parameters such as frequencyof operation. Then select an assembly method corresponding to a workablemounted height for the components having the highest power rating. Thenselect an assembly method corresponding to the same or a lower mountedheight for components having the next highest power rating, and so onuntil all the system components have been accounted for. It may benecessary to iterate the procedure if the lower powered devices end upwith a greater mounted height than higher powered devices. Thisprocedure will provide an optimized heat-sinking strategy with respectto component power. In embodiments of the present disclosure, apreferred height in the range of 0.5-2.5 mm for example will make almostno difference to the cooling performance. It will be appreciated thatthe preferred height may be less than 0.5 mm or greater than 2.5 mm.This insensitivity of cooling performance to preferred height is becausethe back face of components is already disposed as closely as possibleto a heat sinking surface, independently of the preferred height.However, a lower preferred height will result in a more densely packedelectronic system having potentially a higher power density inembodiments of the present disclosure.

A worldwide infrastructure exists for semiconductor packaging. There areover 120 OSAT (Outsourced Semiconductor Assembly and Test) companies andover 360 packaging facilities worldwide. Accordingly, it may be possibleto use multiple sources for flip chip bumping and flip chip assembly,surface mount assembly, interposers, chiplets and embedded bridges asdescribed herein.

Regarding potential problems arising from thermal expansion effects, thefollowing thermal expansion coefficients are typical: silicon 2.6×10⁻⁶/°K; copper 17×10⁻⁶/° K; FR-4 11×10⁻⁶/° K (lengthwise); KAPTON 20×10⁻⁶/°K; alumina (a common substrate material for SMDs) 4.5-11×10⁻⁶/° K;SYLGARD 184 filler 340×10⁻⁶/° K. With respect to the interface betweenthe rear face of a mounted component and a heat sinking surface, the DAFis formulated to handle significant die shear, >2,000 psi forESP7666-HK-DAF. Considering the mix of materials in a printed circuitboard assembly 56, the stresses due to thermal expansion and contractionmay be moderate, and heat curing of the filler material may help torelieve stresses incurred during prior assembly steps. Additionalannealing steps may further reduce stress in embodiments of the presentdisclosure. SYLGARD 184 has a durometer of ShoreA 43, representing asoft and compressible material; this may mitigate its high value ofthermal expansion coefficient.

The sealed nature of blade server embodiment 50 has advantages in termsof robustness and reliability; however, it may be inherently difficultto repair. For maintainability it may be advantageous to adopt a systemlevel strategy like one that has evolved for flash memories. i.e.provide redundant devices, prepare and maintain a map of the good andbad devices, swap out any malfunctioning devices at the testing stage,and optionally monitor the health of all devices during operation toswap out any devices that have malfunctioned.

Regarding the distribution of power in a printed circuit board assemblyof the present disclosure, it may be desirable to regulate power locallyusing, for example, either power-related bare die or power-related SMDs.The advantageous cooling characteristics of the proposed printed circuitboard assemblies may enable higher levels of power dissipation than iscustomary in power-related components.

FIG. 6 illustrates the layout 66 of packaged processors 67 and dualinline memory modules (DIMMs) 68 on the printed circuit board 69 of theHP PROLIANT BL460cG8 blade server 20, an example of prior art. Thedrawing is approximately at ¼ scale. Each DIMM contains 9 DDR3 memorychips, each having a die area of 35 mm². The packaged processors occupya large fraction of the board space, and this is primarily driven bypower consumption. Each XEON E5-2660v4 processor has a thermal designpower (TDP) of 105 W, and the peak power may be 20-50% higher.Processors such as these may be designed to slow down if a junctiontemperature of around 150° C. is exceeded. As previously noted, thenumber of processor chips per unit system volume is 2/319 in³=0.0063processors/in³.

FIG. 7 illustrates a possible board layout of a printed circuit boardassembly 70, an embodiment of the present disclosure, using bare dierather than packaged devices. The spaces provided between bare die areestimates only. The same board size 69 and the same processors andmemories used in the HP PROLIANT BL460cG8 blade server are used. EachXEON E5-2660v4 processor die 71 has an area of 306.2 mm². Each DDR3memory chip 72, K4B2G0846D-HCH9, has an area of 35 mm² and a powerconsumption of around 2.5 W. In one embodiment seven processor die 71may be placed on each side of PCB 69. The number of processors per unitsystem volume=14/(1.1×7.1×20.4)=0.089 processors/in³. The layout shows37×8=296 memory die (per side), or 592 total memory die compared with144 total die in the sixteen DIMMs 68 of FIG. 6. The total powerdissipation of processors and memory per unit system volume calculatesas [(14×105 W)+(592×2.5 W)]/159 in³=16.6 W/in³.

FIG. 8 shows an alternative board layout of a printed circuit boardassembly 80, another embodiment of the present disclosure. The sameboard size 69 and the same processors and memories used in the HPPROLIANT BL460cG8 blade server are used. In one embodiment, fiveprocessor chips 71 may be provided on each side of PCB 69, andsixty-four memory die 72 may be provided per processor, as shown in FIG.8, for a total of 640 memory die per printed circuit board assembly 80.With this layout, the circuit paths connecting processor chips 71 tomemory chips 72 are shorter than those in the HP PROLIANT BL460cG8 bladeserver, and this may lead to higher performance. The total number ofprocessor chips per unit system volume=10/(1.1×7.1×20.4)=0.063.

For higher component and assembly yield, it may be advantageous in aserver application to use server chiplets, each server chipletcomprising a processor chip plus a large number of memory chips forexample. These chiplets can be tested and validated as high-levelcomponents prior to assembly into a PCBA.

The thermal design of the blade server embodiment 50 is now considered.The primary thermal advantage of the proposed bare die configuration isthat, at least for the highest power components, the thermal path fromeach component to cooling water comprises only a thin sheet of dieattach film, (DAF) in series with a sheet of copper (a cooling tank wallfor example). The best case occurs when a predetermined preferred heightis used for a mounted component. If a lower mounted height is used, thenthe thermal resistance of filler material must be considered, asdetailed in reference to FIG. 15. Filler material may be air or SYLGARD184 or ESP7666-HK-DAF as non-limiting examples. Silicon has a thermalconductivity of 149 W/m° K. Copper has a thermal conductivity of 390W/m° K. Air has a thermal conductivity of 0.028 W/m° K. SYLGARD 184 hasa thermal conductivity of 0.27 W/m ° K. Die attach film ESP7666-HK-DAF63 a, 63 b depicted in FIG. 5 and available from Dow has a thermalconductivity of 1.8 W/m° K. It is available in thicknesses of 20 μm and40 μm. For ease of use, the 40 μm thickness is used in the followingcalculation of θ_(D-W), the thermal resistance of the DAF measuredbetween a die such as 56 a and cooling water 52 a, as shown in FIG. 5.The die area for a XEON E5-2660v4 processor is 236 mm². Thermalresistance θ=t/kA, where t is the thickness in meters, k is the thermalconductivity in W/m° C. and A is the area in m².

In this example wherein the preferred height is used for the mountedcomponent, a silicon die thickness of 775 μm is assumed, and a wallthickness of copper tank 51 a is assumed at 1.5 mm:

$\begin{matrix}{\theta_{{D\;\underset{\_}{1}} - W} = {\theta_{{D\;\underset{\_}{1}} - {D\;\underset{\_}{2}}} + \theta_{{D\;\underset{\_}{2}} - {Cu}} + \theta_{{Cu} - W}}} \\{= {{775 \times 10^{- 6}m^{2}{{{^\circ}K}/\left( {149W \times 236 \times 10^{- 6}m^{2}} \right)}} +}} \\{{40 \times 10^{- 6}m^{2}{{{^\circ}K}/\left( {1.8W \times 236 \times 10^{- 6}m^{2}} \right)}} +} \\{1.5 \times 10^{- 3}m^{2}{{{^\circ}K}/\left( {390W \times 236 \times 10^{- 6}m^{2}} \right)}} \\{= {{\left( {0.022 + 0.094 + 0.016} \right){^\circ}\mspace{14mu}{{C.}/W}} = {0.132{^\circ}\mspace{14mu}{{C.}/{W.}}}}}\end{matrix}$

Power dissipation P in W between surfaces ΔT° C. apart in temperatureand having a thermal resistance of θ between them is:P=ΔT/θ.Assuming a conservative maximum die temperature for the processors of120° C. and assuming the cooling water has a maximum temperature of 40°C., then ΔT equals 80° C. and P=80/0.132=606 W. The high cooling marginin this example may enable the use of higher power chips. Using the PCBAlayout of FIG. 7 and employing seven processors per side, the maximumpower dissipation using XEON E5-2660v4 processors calculates as 14×105W=1,470 W. Assuming blade server dimensions of 1.1×7.1×20.4=159/in³, theprocessor power density in the blade server embodiment 50 is1,470/159=9.2 W/in³ compared with 210/318=0.66 W/in³ for the HP PROLIANTBL460cG8, an increase of around 14×. Assuming fourteen processors and1,600 W total power dissipation in the blade server embodiment, therequired water flow rate is calculated to be approximately 0.076 gallonsper minute or 0.29 liters per minute.

FIG. 9 illustrates a PCBA 90 in an embodiment of the present disclosure.Filler material 61 is shown on all sides, after mounting components oneach side of a PCB 94, and planarizing each side with the filler 61. Afirst preferred thickness t₁, 92, is shown for component heights on afirst side of PCB 94. A second preferred thickness t₂, 93, is shown forcomponent heights on a second side of PCB 94. The preferred thicknessest₁ and t₂ may be the same, or different as shown. Components may bemounted on both sides of PCB 94 as shown in FIG. 9, or on one side only.Similarly, filler 61 may be provided on one or both sides of PCB 94. PCB94 is shown as a flex circuit having traces 95 that may be coupled toterminals of a connector for accessing external signals and power. Thevertical scale in FIG. 9 is expanded for illustration purposes. Detail96 is an expanded view of the upper surface of PCBA 90. Detail 96 showsa component 97 with no covering of filler material; accordingly, it willhave the lowest thermal resistance and the best cooling performance.Component 97 may be one of the chips in PCBA 90 having a high-powerconsumption. Component 98 is shown with a thin covering of filler 61; itmay have a lower power consumption and may have a lower assembledheight. The thermal resistance seen by component 98 will be greater thanthe thermal resistance seen by component 97, due to the covering offiller material 61. However, because this chip has a lower powerdissipation it will be adequately cooled in one of the embodimentsdescribed herein.

Having discussed a blade server embodiment, a larger scale electronicsystem will now be described. FIG. 10 illustrates a water-cooledelectronic system 99 in another embodiment of the present disclosure.Cooling fluids other than plain water may be employed, such as a mixtureof water and ethylene glycol. Electronic system 99 is enclosed in a tankenclosure 99 a having exemplary dimensions of 19 inches wide, 17.5inches high and 36 inches deep. A front panel connector 99 b is shown,for connecting external signals and power. Water inlet (or outlet) ports99 c are also shown.

FIG. 11 illustrates electronic system 99 as a cross-section that islabeled AA in FIG. 10. Laminate blocks 101 a and 101 b are shownsubstantially immersed in water 102 inside tank enclosure 99 a. Thelaminate blocks rest on a baseplate 103 which may be carried by supportblades 104. An inner tank 105 is shown. Arrows 106 a,b,c,d indicatewater flow along longitudinal flow channels provided within tankenclosure 99 a. Laminate blocks 101 a and 101 b comprise laminations ofprinted circuit assemblies and copper foils, detailed in FIG. 12. Eachprinted circuit board assembly includes a flexible printed circuit board(flex circuit) having traces indicated by dashed line 107 that connectwith a connector 108 whose terminals are coupled to correspondingterminals of front panel connector 99 b of FIG. 10.

FIG. 12 includes an expanded view of a lamination structure depicted inFIG. 11. A lamination 110 is shown, including a printed circuit boardassembly (PCBA) assembled on a flexible substrate 107 and having twobranches 111 a and 111 b. Bare die components such as a processor die112 are shown flip chip mounted on flexible substrate 107. As previouslydescribed, an interposer may be required to redistribute the thousandsof points of input/output typically required for a powerful processorchip. Also as previously described, flip chip terminals may be copperpillars such as 109 a or solder balls such as 109 b, or any other typeof flip chip terminal. Parallel sheets of metal foil 113 are showninterleaved with branches 111 a and 111 b of the PCBA to form repeatedlaminations 110. As detailed in reference to FIG. 5, assembledcomponents may include bare die, SMDs and stacked devices. The assembledcomponents have a preferred height as previously described. Fillermaterial 61 is also shown, and it has enabled PCBA branches 111 a and111 b to be planarized as shown. Die attach film (DAF) 63 a is shown,coupling each component to a metal foil 113. Metal foil 113 may comprisea 16 oz copper foil having a thickness of 0.022 inches (0.56 mm) forexample, although foil or sheet material of any metal composition andany suitable thickness may be used. Printed circuit board 107 may be aflex circuit as shown and may be fabricated on a KAPTON substrate forexample. Other printed circuit board materials and configurations may beused. The base of each copper foil 113 may be soldered 114 to baseplate103 as shown. Base plate 103 may be soldered 115 to blade 104 thatconnects with the tank enclosure 99 a as shown.

In the lamination 110 of FIG. 12, the thermal resistance from the frontface 112 a of a die such as 112 to an associated copper foil surfacesuch as 113 b is calculated. An assumption is made about the maximumtemperature of the copper foil, given the thermal path from an interiorpoint on the foil to the cooling fluid. Again, assume a XEON E5-2660v4processor having a die thickness of 775 um and an area of 236 mm². Alsoassume a 40 μm thick layer of die attach film ESP7666-HK-DAF 58.

$\begin{matrix}{\theta_{{D\;\underset{\_}{1}} - W} = {\theta_{{D\;\underset{\_}{1}} - {D\;\underset{\_}{2}}} + \theta_{{D\;\underset{\_}{2}} - {Cu}}}} \\{= {{775 \times 10^{- 6}m^{2}{{{^\circ}K}/\left( {149W \times 236 \times 10^{- 6}m^{2}} \right)}} +}} \\{{40 \times 10^{- 6}m^{2}{{{^\circ}K}/\left( {1.8W \times 236 \times 10^{- 6}m^{2}} \right)}} +} \\{= {{\left( {0.022 + 0.094 + 0.016} \right){^\circ}\mspace{14mu}{{C.}/W}} = {0.132{^\circ}\mspace{14mu}{{C.}/{W.}}}}}\end{matrix}$

Assume that the hottest interior portion of copper foil in a laminateblock such as 101 a is at 80° C., 40° C. higher than the temperature ofthe cooling water. If, for a particularly aggressive cooling schemethermal modeling reveals that the interior portions of a laminate blockwill get too hot, two remedies may be considered: (i) making thelaminate blocks thinner and positioning water cooling tanks betweenthem, or (ii) increasing the thickness of the copper foils. Assuming amaximum die temperature of 150° C., ΔT is calculated as 150−80=70° C.The maximum power dissipation permitted per processor is P=ΔT/θ,=70/0.116=603 W. Although approximate, this again represents a highcooling margin for the assumed XEON E5-2660v4 processor.

FIG. 13 shows a possible layout 120 of a processor group comprising aprocessor chip 121 assembled with 76 memory die 122 dedicated to theprocessor. The memory die may be K4B2G0846D-HCH9 (DDR3) chips having anarea of 35 mm² for example. Using the XEON E5-2660v4 processor and theDDR3 memory chips, the area 123 of the processor group including the 76memory chips approximates 112 mm×87 mm or 9,744 mm². The total powerconsumption for the group may be 105 W for the processor chip and 2.5 Wfor each memory chip, for a total of 295 W. An alternative to assemblyof each individual die in the processor group (one at a time) is tobuild a chiplet having the same functionality, wherein the chiplet isassembled and tested as a single component. Different chiplets thatcombine many different chip functions may be used. The types of chipsused in chiplets may include processors, memories, power-related chips,communications-related chips, plus chips comprising any other chipfunction. Using wafer or chip thinning together with judicious selectionof the terminal type, e.g. copper pillars or copper pillar bumps orsolder balls, a consistent height may be achieved for all the chips in achiplet. The consistent height for a chiplet is similar in concept tothe preferred height for components mounted in PCBAs of the presentdisclosure, providing for excellent cooling of the chiplet components.

The thickness of a single-branch lamination, as shown in FIG. 12,approximates 2.7 mm. The metal foil thickness may vary from 0.25-2.0 mmfor example, resulting in a total thickness of a single-branchlamination in an approximate range of 2.0-3.7 mm. The useful PCBA areaper single-branch lamination approximates 328,000 mm² per side. Thus,the total number of processor groups per single lamination approximatelyequals 328,000/9,744×2 sides=68. The thickness of each block may be 6.5inches or 165 mm for example. The thickness of each lamination at 2.7 mmleads to 165/2.7×2 blocks=122 laminations total in electronic system 50.Thus, the total number of processor groups per electronic system equalsapproximately 122×68=8,296. The total power in the electronic system,using 105 W processor chips and 2.5 W memory chips approximately equals295 W×8,296=2.45 MW. The flow rate of cooling water required to maintainΔT of 70° C. approximately equals 133 gallons per minute. The density ofprocessors per unit system volume is calculated as 8,296/11,970=0.69processors/in³. The total power dissipation per unit systemvolume=2.45×10⁶ W/11,970 in³=205 W/in³.

FIG. 14 is a flow chart of a method 140 for manufacturing and deployingan electronic system such as electronic system 99 depicted in FIGS.10-12, according to an embodiment of the present disclosure. Method 140begins with fabricating an inner structure within an outer tank whereinthe inner structure comprises at least one laminate block and eachlaminate block comprises a repeated lamination of a printed circuitboard assembly and a metal foil, step 141. The method continues with,for each repeated lamination, coupling the metal foil to a heat sinkingsurface, step 142. The final step of the method is circulating a liquidcoolant in passages provided between the inner structure and the outertank, to include flowing liquid coolant past the heat sinking surface,step 143.

A further method is described for manufacturing an electronic system inan embodiment of the present disclosure. The method begins withfabrication of a plurality of flexible PCBs having a top edge, a bottomedge, and two end edges. The method continues with selecting a firstplurality of components having approximately a first preferred mountedheight to be mounted on a first side of the plurality of flexible PCBs.The method continues with selecting a second plurality of componentshaving approximately a second preferred mounted height to be mounted ona second side of the plurality of flexible PCBs. The method continueswith mounting the first and second plurality of components on the firstand second sides of the flexible PCB to form a plurality of printedcircuit board assemblies (PCBAs). The method continues with overlaying aco-extensive die attach film atop the first and second plurality ofcomponents on each side of the PCBAs. The method continues with sizingsheets of metal foil to be co-extensive with the PCBAs except slightlyretracted at a top edge, and slightly extended at the bottom edge andtwo end edges. The method continues with overlaying a sized sheet ofmetal foil atop the die attach film on each side of each of theplurality of PCBAs to form a plurality of laminate structures, whereinthe top edge of the metal foil is slightly retracted compared with thetop edge of each of the PCBAs and slightly extended compared with thebottom edge and two end edges of each of the PCBAs. The method continueswith aligning and assembling the plurality of laminate structures intoone or more laminate blocks. The method continues with heating the oneor more laminate blocks to achieve melt-flow of the die attach films.The method continues with cooling the one or more laminate blocks. Themethod continues with applying solder paste to the three extended edgesof the copper foil in each of the one or more laminate blocks. Themethod continues with positioning the one or more laminate blocks on abase plate. The method continues with heating the one- or more laminateblocks positioned on the base plate to achieve melt-flow of the solderpaste and joining of the bottom edge of the copper foil to the baseplate and joining of each of the two end edges to an end plate thatseals an end of the one or more laminate blocks, preventing waterintrusion. The method continues with connecting traces of the PCBA atthe extended top edge to a block connector configured for each laminateblock. The method finishes with coupling terminals of each laminateblock connector to corresponding terminals of a front panel connector ora rear panel connector.

As a measure of computational density, the number of processors per unitsystem volume as described herein are summarized in Table 1. In eachcase the processor is a XEON processor running at 2.2 GHz.

TABLE 1 System Comp. Density Advantage Cray XC040 supercomputer 0.0016 1X HP Proliant BL460cG8 Blade Server 0.0063  3.9X Blade Server 50 0.063 39X Electronic System 90 0.69 431X

Table 1 indicates the effectiveness of using bare die components orstacked bare die components instead of conventionally packaged die, plusthe benefit of a densely packed internal structure integrated with watercooling.

A XEON E5-2660v4 processor chip has been shown to have an availablepower dissipation of 606 W when flip chip mounted as a bare die havingthe preferred height and no intervening filler material (except for adie attach film) between the back face of the die and a heat sinkingsurface. The back face of the die is bonded to a wall of a water filledcopper tank using die attach film ESP7666-HK-DAF as illustrated in FIG.5. The effect of mounting this component at less than the preferredheight is shown in FIG. 15 where the available cooling power in watts isplotted against filler thickness in microns for two filler materials andfor air. The filler materials are disposed in the assembly gap betweenthe back face of the chip and the heat sinking surface. For each curve a40μ thickness of DAF is assumed in series with the named filler. Curve151 applies to the use of die attach material ESP7666-HK-DAF as a fillermaterial, wherein multiple sheets of the material may be stacked toachieve a desired thickness. Curve 152 applies to the use of SYLGARD 184as a filler material. Curve 153 applies to the use of air as a fillermaterial. Other filler materials may be used, especially those with highthermal conductivity. It is desirable that air bubbles be removed; thismay be achieved using a vacuum process, applied while the filler is inliquid form for example. Filler thickness is a measure of the differencebetween an actual component height and the preferred height.

FIG. 15 shows that the choice of filler material is critical, and thatair is a poor choice. Since the cooling performance varies dramaticallywith filler thickness, and the desired thicknesses may be difficult toachieve in practice, FIG. 15 also shows that assembly precision isrequired in order to reap the benefits of a preferred height strategydescribed herein. For good cooling performance a component must bemounted at the preferred height or close to it, as shown in the graph.While the mounting height may be non-critical for a low-power devicelike a dynamic RAM with a rated power of around 2.5 W for example, beingclose to the preferred height is critical for high-power components. Forexample, any component of a PCBA having a power rating of at least 50 Wmay have a mounted height in the range of 90-100% of the preferredheight. To help achieve the desired assembly precision, excess fillermaterial may be applied, then material removed using a grinding processor a polishing process or a chemical mechanical polishing (CMP) processor a combination of these processes, until the back face of thehighest-power components is exposed, and a polished planar surface ofthe PCBA is available for bonding to a heat sinking surface. In additionto removal of filler material, semiconductor material may also beremoved from the back side of high-powered components, to overcome anysmall differences in mounted height (due to assembly tolerances forexample), or any planarity variations, again due to assembly tolerancesfor example.

In embodiments of the present disclosure chiplets may be fabricated inaccordance with a preferred height strategy determined for a host PCBA.Techniques including filling, grinding and polishing and removal ofsemiconductor material may be applied to the construction of chiplets,as described herein for PCBAs, according to embodiments of the presentdisclosure.

As will be understood by those familiar with the art, the invention maybe embodied in other specific forms without departing from the spirit oressential characteristics thereof. Likewise, the particular naming anddivision of the members, features, attributes, and other aspects are notmandatory or significant, and the mechanisms that implement theinvention or its features may have different structural construct,names, and divisions. Accordingly, the disclosure of the invention isintended to be illustrative, but not limiting, of the scope of theinvention.

While the invention has been described in terms of several embodiments,those of ordinary skill in the art will recognize that the invention isnot limited to the embodiments described but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. Another embodiment may comprise air as a coolant fluid forexample. The description is thus to be regarded as illustrative insteadof limiting. There are numerous other variations to different aspects ofthe invention described above, which in the interest of conciseness havenot been provided in detail. Accordingly, other embodiments are withinthe scope of the claims.

The invention has been described in relation to particular examples,which are intended in all respects to be illustrative rather thanrestrictive. Those skilled in the art will appreciate that manydifferent combinations will be suitable for practicing the presentinvention. For example, assembly details for a PCBA of the presentdisclosure may be applied to either a blade server or an electronicsystem of the present disclosure. Other implementations of the inventionwill be apparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. Variousaspects and/or components of the described embodiments may be usedsingly or in any combination. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

What is claimed is:
 1. A printed circuit board assembly (PCBA)comprising: a substrate; a plurality of electronic components assembledon the substrate using flip chip assembly methods, wherein the pluralityof electronic components are selected from bare die, surface mountdevices, and stacked devices; a filler disposed between the plurality ofelectronic components; wherein a preferred height is predetermined forat least one of the plurality of electronic components, the preferredheight corresponding to one or more of the plurality of electroniccomponents having a high power rating; wherein the filler and theplurality of electronic components are ground or polished to form apolished planar surface having a height equal to the preferred height.2. The printed circuit board assembly of claim 1, wherein the stackeddevices comprise an interposer, a chiplet, or an embedded bridge.
 3. Theprinted circuit board assembly of claim 1, wherein the substratecomprises traces that are coupled to corresponding terminals of a PCBAconnector.
 4. The printed circuit board assembly of claim 1, furthercomprising: a metal member having a length and a width approximatelyco-extensive with a length and width of the printed circuit boardassembly; thermal interface material coupled to the metal member;wherein at least one of the plurality of electronic components comprisesat least one high-power component, and wherein a back face of the atleast one high-power component is coupled to the metal member using thethermal interface material to form a lamination of the printed circuitboard assembly and the metal member.
 5. The printed circuit boardassembly of claim 1, wherein the preferred height is 2.5 mm.
 6. Aprinted circuit board assembly (PCBA) comprising: a substrate; aplurality of electronic components assembled on the substrate using flipchip assembly methods, wherein the plurality of electronic componentsare selected from bare die, surface mount devices, and stacked devices;wherein a preferred height is predetermined for at least one of theplurality of electronic components, the preferred height correspondingto one or more of the plurality of electronic components having a highpower rating; a metal member having a length and a width approximatelyco-extensive with a length and width of the printed circuit boardassembly; thermal interface material coupled to the metal member;wherein at least one of the plurality of electronic components comprisesat least one high-power component, and wherein a back face of the atleast one high-power component is coupled to the metal member using thethermal interface material to form a lamination of the printed circuitboard assembly and the metal member.
 7. The printed circuit boardassembly of claim 6, wherein the thermal interface material comprises adie attach film.
 8. The printed circuit board assembly of claim 6,wherein the thermal interface material has a thickness less than 50microns.
 9. The printed circuit board assembly of claim 1, wherein thepreferred height is in the range of 0.5-3.0 mm.